Method for automatically capturing data from non-networked production equipment

ABSTRACT

One variation of a method for automatically capturing data from non-networked production equipment includes: detecting a location of a mobile device within a facility, the mobile device manipulated by an operator while performing a step of an augmented digital procedure at a machine in the facility; estimating a position of a display on the machine relative to a field of view of an optical sensor in the mobile device based on the location of the mobile device and a stored location of the machine within the facility; in response to the position of the display falling within the field of view of the optical sensor, selecting an image captured by the optical sensor; extracting a value, presented on the display, from a region of the image depicting the display; and storing the value in a procedure file for the augmented digital procedure completed at the machine.

CROSS-REFERENCE TO RELATED APPLICATIONS

This Application is a continuation application of U.S. patentapplication Ser. No. 16/700,851, filed on 2 Dec. 2019, which claims thebenefit of U.S. Provisional Application No. 62/773,763, filed on 30 Nov.2018, each of which is incorporated in its entirety by this reference.

This Application is a continuation-in-part of U.S. patent applicationSer. No. 16/678,992, filed on 8 Nov. 2019, which is incorporated in itsentirety by this reference.

BRIEF DESCRIPTION OF THE FIGURES

FIG. 1 is a flowchart representation of a method;

FIG. 2 is a flowchart representation of one variation of the method;

FIG. 3 is a flowchart representation of one variation of the method;

FIGS. 4A and 4B are graphical and flowchart representations of onevariation of the method;

FIGS. 5A, 5B, and 5C are flowchart representations of variations of themethod;

FIG. 6 is a flowchart representation of one variation of the method; and

FIG. 7 is a flowchart representation of one variation of the method.

DESCRIPTION OF THE EMBODIMENTS

The following description of embodiments of the invention is notintended to limit the invention to these embodiments but rather toenable a person skilled in the art to make and use this invention.Variations, configurations, implementations, example implementations,and examples described herein are optional and are not exclusive to thevariations, configurations, implementations, example implementations,and examples they describe. The invention described herein can includeany and all permutations of these variations, configurations,implementations, example implementations, and examples.

1. Method

As shown in FIG. 1, a method S100 for automatically capturing data fromnon-networked production equipment includes: detecting a first locationof a mobile device within a facility at a first time, the mobile devicemanipulated by an operator while performing a step of an augmenteddigital procedure at a machine in the facility in Block S110; estimatinga first position of a display on the machine relative to a field of viewof an optical sensor in the mobile device based on the first location ofthe mobile device and a stored location of the machine within thefacility in Block S120, the step in the augmented digital procedurespecifying data capture from the display; in response to the firstposition of the display falling within the field of view of the opticalsensor at the first time, selecting a first image captured by theoptical sensor at approximately (e.g., within one second, within tenseconds) the first time in Block S130; extracting a first value,presented on the display at approximately the first time, from a firstregion of the first image depicting the display in Block S142; andstoring the first value in a procedure file for the augmented digitalprocedure completed at the machine in Block S150.

One variation of the method shown in FIGS. 2 and 3 includes: accessing avideo feed captured by an optical sensor in a mobile device manipulatedby an operator while performing a step of an augmented digital procedureat a machine in the facility in Block S110; accessing a definition of afeature corresponding to a display coupled to the machine, the step inthe augmented digital procedure specifying data capture from the displayin Block S162; scanning the video feed for the feature in Block S146; inresponse to detecting the feature in a first image in the video feed,detecting the display in a first region of the first image in BlockS140; extracting a first value, presented on the display, from the firstregion of the first image in Block S142; and storing the first value ina procedure file for the augmented digital procedure completed at themachine in Block S150.

Another variation of the method shown in FIG. 6 includes: accessing avideo feed captured by an optical sensor in a facility in Block S132;selecting an image in the video feed in Block S130; detecting a displayon a machine depicted in a region of the image in Block S140;identifying the machine depicted in the image in Block S160; retrievinga data extraction model, from a set of data extraction models, linked tothe display and the machine in Block S162; extracting a value from theregion of the image based on the data extraction model in Block S142;and storing the value in a procedure file associated with the machine inBlock S150.

2. Applications

The method S100 can be implemented within a production facility in orderto reduce friction for recording data from non-networked (or “siloed”)equipment within the facility and to support operator interfacing with,managing, and operating this equipment. In particular, the method S100can be executed by a mobile device (and/or a fixed imaging system)—suchas in conjunction with a remote computer system—within a pharmaceuticalproduction facility to automatically detect displays (e.g., digitaldisplays, analog dials, scales, readouts, valve orientations, analogthermometers) on a machine, read data from these displays, verify thesedata, and store these verified data as a human operator or technicianwearing the mobile device (e.g., in the form of an augmented realityheadset) or carrying the mobile device (e.g., in the form of asmartphone or tablet) performs steps within a predefined test oroperating procedure at this machine. Therefore, by automaticallydetecting these displays in the field of view of a camera in the mobiledevice, automatically reading values from these displays, andautomatically storing these data—such as in a procedure file specific tothis machine and instance of the augmented digital procedure—accordingto Blocks of the method S100, the mobile device can free the operator tofocus on performing actions specified in steps of the procedure andreduce opportunity for human error that may occur during manual dataentry (e.g., from multiple data entries of the same information, fromdata entry of values in an unexpected sequence).

In addition to automatically recording data from manufacturing equipmentthat may be intentionally non-networked, such as to meet securityrequirements for pharmaceutical production equipment, the mobile deviceexecuting the method S100 can present steps of a procedure and guidancefor performing these steps to the operator, such as in the form ofaugmented reality content including visual prompts and/or virtualoverlays for aligning a camera in a mobile device to a display on amachine designated for data capture during a step of the procedure.

The method S100 can therefore be implemented within a productionfacility in order to guide an operator through manual completion ofsteps within a procedure at such a machine and to automatically recorddata—presented on human-vision-optimized displays arranged on thismachine—on behalf of the operator.

3. System

Generally, Blocks of the method S100 can be executed by a systemincluding: a computer system, such as a remote server or a computernetwork; and a mobile device, such as including or connected to anaugmented-reality headset. For example, the mobile device can be anaugmented reality headset, including a heads-up display, eyes-updisplay, head-mounted display, or smart glasses configured to renderaugmented reality content for an operator wearing this mobile device, asshown in FIG. 6.

Furthermore, the mobile device can include: a suite of sensorsconfigured to collect information about the mobile device's environment;local memory configured to temporarily store a localization map of aroom; and a controller configured to determine a location of the mobiledevice in real space, such as based on the localization map and datacollected by the suite of sensors. For example, the mobile device caninclude: a depth camera paired with a 2D color camera; or a pair ofstereoscopic 2D color cameras. Each of these optical sensors can outputa video feed containing a sequence of digital photographic images (or“frames”), such as at a rate of 20 Hz, and the controller can compileconcurrent frames output by these optical sensors into a 3D point cloudor other representation of surfaces or features in the field of view ofthe mobile device. Following receipt of a localization map of a roomoccupied by the mobile device and generation of a 3D point cloud (orother representation of surfaces or features in the field of view of themobile device), the controller can implement point-to-plane fitting orother techniques to calculate a transform that maps the 3D point cloudonto the localization map in order to determine the pose of the mobiledevice within the room.

However, the mobile device can include any other type of sensor in anyother quantity and can implement any other method or technique tocalculate its pose within a room based on a localization map of the roomand data recorded by these sensors.

4. Procedure Augmentation

The remote computer system can interface with an administrator throughan administrator portal to construct an augmented digital proceduredefining a set of steps and specifications for recording data during aprocedure at a particular machine or make, model, and/or configurationof machine within the facility.

4.1 Existing Procedure Augmentation

In one implementation shown in FIGS. 2, 4A, and 4B, an administratoraffiliated with the facility accesses an administrator portal, accessesa procedure database, and loads a digital (e.g., vectorized, digitized)copy of a document—outlining steps of a procedure for a machine in thefacility—from the procedure database into the administrator portal.(Alternately, the administrator can scan a paper copy of this document,such as with a smartphone, tablet, or dedicated scanner, and then loadthis digital copy into the administrator portal).

4.2 Procedure Identifier

The administrator portal can then interface with the administrator to:highlight a procedure identifier in the digital copy of this document,such as a QR code, barcode, alphanumeric procedure identifier andrevision number, or textual description of the procedure; and link thisprocedure identifier to a particular machine, type or class of machine,or configuration of machine in the facility and/or to a particularlocation, room, or area inside the facility. For example, theadministrator can select each machine, machine type or class, or machineconfiguration from a dropdown menu—rendered in the administratorportal—of all machines in the facility and/or select a machine orlocation within the facility from a map (e.g., a plan map, or a 3Dlocalization map) of the facility—rendered in the administratorportal—to link to this procedure identifier. The administrator portalcan similarly interface with the administrator to link supportequipment, such as a scale, to this procedure identifier.

Alternatively, the administrator portal (or a remote computer system)can automatically read machine and/or support equipment identifiers fromthis digital copy and link equipment within the facility to thisidentifier accordingly.

4.3 Input Fields

The administrator portal can further interface with the administrator tolink input fields to steps in the digital copy and to assign data typesto these input fields. In one example, the administrator selects a stepin the digital copy and selects a data type from a list or menu of datatypes, such as including: “voice capture”; “manual note capture”;“manual numerical value capture”; “manual image capture”; and/or“automatic data capture.”

For example, for a step in the procedure assigned a “voice capture” datafield, a mobile device can record an audio snippet via a microphone inthe mobile device when triggered by the operator or automatically whenthe operator is performing this step within an instance of the augmenteddigital procedure. For a step in the procedure assigned a “manualnumerical value capture” data field, the mobile device can record anumerical value entered manually by the operator via a virtual keypadrendered on the mobile device as the operator performs this step withinthis instance of the augmented digital procedure. For a step in theprocedure assigned a “manual note capture” data field, the mobile devicecan record an alphanumeric text string entered manually by the operatorvia a virtual keyboard rendered on the mobile device as the operatorperforms this step within this instance of the augmented digitalprocedure.

However, for a step in the procedure assigned an “automatic datacapture” data field, the mobile device can automatically record an imageor a video snippet via the camera in the mobile device when a display(e.g., a digital display or readout; an analog dial) depicting datarelevant to the step falls within the field of view of the camera, suchas while the operator performs this particular step within an instanceof this procedure. The mobile device (or the remote computer system) canthen automatically extract a relevant value from this image or videosnippet according to a type of the corresponding input field and storethese data in a procedure file for this instance of the augmenteddigital procedure, as described below.

The administrator can similarly assign input fields and data types toother steps in the procedure document.

4.4 New Procedure Generation

In one variation, rather than augment an existing procedure documentwith input fields and data types, the administrator portal can interfacewith the administrator to generate a new procedure for a machine in thefacility. For example, the administrator portal can initialize a new(empty) digital procedure document. The administrator can then: type orcopy a textual description of a step within the procedure into a stepfield in the new digital procedure document; enable an input field forthis step; and select a data type for this input field, such as from adropdown menu of data types; and repeat this process for other steps inthe procedure. Alternatively, an administrator may build a procedure,such as by generating the procedure as an expert operator is tracked,including annotating steps performing by the expert operator during theprocedure and augmenting these annotated steps with photo, video, audio,and/or 3D capture files recorded throughout this instance of theaugmented digital procedure.

The administrator portal (or the remote computer system) can thencompile these steps, input fields, and data types for these input fieldsinto an augmented digital procedure.

4.5 Automatic Image Capture Triggers

The administrator portal can then interface with the administrator todefine a trigger for automatic data capture for a step in the proceduredocument containing a data field designated for “automatic datacapture.”

In one implementation, the administrator portal can interface with theadministrator to further enhance the augmented digital procedure withaugmented reality content, as shown in FIG. 1. In one implementation,when configuring the augmented digital procedure, the administrator canaccess a localization map of the facility (as described above), such asincluding representations of: machines and equipment deployed throughoutthe facility; analog or digital controls on these machines andequipment; and locations of displays on these machines and equipment.The administrator can then: link the augmented digital procedure to aparticular machine in the localization map; select a step of theaugmented digital procedure specifying a control input into the machine;and link this step to a corresponding control—on the machine—representedin the localization map. Similarly, the administrator can: select a stepof the augmented digital procedure specifying data capture; and linkthis step to a corresponding display—on the machine—represented in thelocalization map. Furthermore, the administrator can: select a step ofthe augmented digital procedure specifying interaction with supportequipment (e.g., a scale); and link this step to a nearest location inthe facility in which such support equipment is stored according to thelocalization map and/or link this step to an existing 3D model of thissupport equipment.

Thus, in this implementation, the administrator portal (or the remotecomputer system) can link a step of the procedure specifying manual orautomatic data capture to a 3D location of the corresponding display ona machine or support equipment arranged in a fixed location within thefacility. For example, the administrator portal (or the remote computersystem) can define this 3D location within a coordinate system definedwithin the facility or within the room occupied by the machine. Later,when an operator wearing or carrying a mobile device performs this stepof the procedure, the mobile device can: track its location andorientation with the facility based on features detected in a videostream recorded by a camera in the mobile device, as described below;calculate a 3D volume in the facility within the field of view of thecamera in the mobile device by projecting a primary axis and an angle ofview of the camera into the localization map according to the positionand orientation of the mobile device; and flag a last frame or sequenceof frames—recorded by the camera—for data extraction if the 3D locationof the display linked to this step currently falls within the field ofview of the camera.

The administrator portal (or the remote computer system) can also:access a virtual model (e.g., a set of template images, a representativeconstellation features, a parametric model) of the display on themachine or support equipment; and further link this step of theprocedure to this virtual model. For example, the display can include:an analog rotary dial; an analog rotary or linear gauge; an analogscale; an analog or digital seven-segment display; a monochrome digitaldisplay; a digital color display (e.g., an LED LCD display); or a CRTdisplay; etc. The administrator portal (or the remote computer system,or the administrator) can thus: select a generic model for a type ofthis display or select a model unique to this display on this make,model, and configuration of machine or support equipment; and link thismodel to this step of the procedure. Later, a mobile device loaded withthis augmented digital procedure (or the remote computer system) canleverage this model to identify this display directly within an imagecaptured by a camera in the mobile device.

Additionally or alternatively, the display or a surface immediatelyadjacent the display can be labeled with an optical fiducial, such as abarcode, QR code, or optical emitter. The administrator portal (or theremote computer system, or the administrator) can then link a model ofthe optical fiducial (e.g., a code contained in the barcode; a virtualcopy of the QR code) to this step. Later, a mobile device loaded withthis augmented digital procedure (or the remote computer system) canleverage this model to identify this optical fiducial directly within animage captured by a camera in the mobile device and then detect theadjacent display in the image accordingly.

The administrator portal (or the remote computer system, or theadministrator) can define additional parameters for image capture anddata extraction for this step of the procedure. In one example, ratherthan present data of a single type (e.g., a current pressure or currenttemperature), the display may scroll or cycle through a set of datatypes (e.g., low pressure in psi, high pressure in psi, and currentpressure in psi at a rate of 0.3 Hz). The administrator portal (or theremote computer system, or the administrator) can thus: access such datapresentation characteristics of the display; and specify capture of oneimage (or a short sequence of three frames) during this step if thedisplay presents data of a single type. However, if the display scrollsthrough data of multiple types, the administrator portal (or the remotecomputer system, or the administrator) can specify capture of a sequenceof images spanning one cycle of the display (e.g., six seconds for adisplay that scrolls through three different values at a rate of 0.3 Hz)during this step. In another example, the display may present valuesthat change substantively on short time scales (e.g., rapidly at a rateof 10 Hz) or on long time scales (e.g., slowly at a rate of 0.1 Hz). Theadministrator portal (or the remote computer system, or theadministrator) can thus: specify capture of one image (or a shortsequence of three frames) during this step if the display presents dataof a single type that changes on a long time scale; and specify captureof a sequence of images (or a longer sequence of twenty frames) duringthis step if the display presents data of a single type that changes ona short time scale (the remote computer system can then compile dataextracted from such as a sequence of frames into one value for thisstep).

Furthermore, the administrator portal (or the remote computer system, orthe administrator) can set minimum resolution requirements capturing animage of the display during this step—such as based on a size of thedisplay generally or sizes of characters or an indicator (e.g., aneedle) on the display—in order to enable the mobile device or remotecomputer system to accurately detect and extract data presented on thisdisplay from an image captured by the mobile device during this step.Later, a mobile device can: calculate a maximum distance from the mobiledevice to the display to achieve a minimum resolution within a region ofan image depicting the display based on known intrinsic and extrinsicproperties of the camera in the mobile device; and then flag an imagefor extraction of a value from this display only if the mobile devicewas located within this maximum distance of the known location of thedisplay when the image was recorded.

The administrator portal (or the remote computer system, or theadministrator) can also link this step in the procedure to a dataextraction model or process for extracting data presented on the displayfrom an image captured by a mobile device during this step, such asdescribed below. However, the administrator, the administrator portal,and/or the remote computer system can define any other parameters forrecording an image of the display during a step of this procedure.

The remote computer system can further interface with the administratorthrough the administrator portal to define parameters for automaticimage capture and data extraction for other steps within the procedure.The computer system can also parameterize this augmented digitalprocedure for application of this procedure and automatic data captureto multiple machines at different physical locations within thefacility. Finally, the remote computer system can compile these stepsand data capture parameters into an augmented digital procedure, asdescribed above, and store this augmented procedure in a digitalprocedure database for later access by the operator.

4.6 Digital Draft Procedure

Additionally or alternatively, rather than interface with theadministrator to manually construct a trigger for automatic data capturewithin a step in the procedure via the administrator portal, the remotecomputer system (or the administrator portal) can: compile stepsextracted from the procedure, input fields defined by the administrator,and data types for these input fields into digital draft procedure;serve this digital draft procedure to a target operator (e.g., anexperience operator) for completion of an exemplary instance of thisdigital draft procedure at a machine in the facility; access timeserieslocation data, a video feed, and/or manual capture data recorded duringthis exemplary instance of the digital draft procedure; and interfacewith the administrator via the administrator portal to transform thesedata (or can implement artificial intelligence or regression techniquesautomatically transform these data) from the exemplary instance of thedigital draft procedure into spatial and/or temporal triggers forautomatic data capture during select steps of this procedure.

In one implementation, the remote computer system (or the administratorportal) accesses a document specifying a set of steps of a procedure foran equipment type of the machine and extracts descriptions of thesesteps from the document, as described above. The administrator portalthen: interfaces with the administrator to link a first description of afirst step in the procedure—extracted from the document—to a first inputfield and to assign a first capture type to this first input field; andrepeats this process for other steps of the procedure specifying datarecordation. The remote computer system then generates a digital draftprocedure containing descriptions of these set steps extracted from theprocedure and linked to input fields specifying capture of data ofparticular capture types, such as described in U.S. Patent ApplicationNo. APPR-M03-US. The administrator portal can also prompt theadministrator to identify a target operator with experience performingthis procedure; and the remote computer system can queue a firstinstance of this digital draft procedure for completion by the targetoperator. In particular, the remote computer system can serve a firstinstance of the digital draft procedure to a mobile device—assigned toor logged into by this target operator—for completion at a machine ofthe equipment type specified in the procedure.

In this implementation, the remote computer system can interface withthe target operator's mobile device to record: images, (2D or 3D) videoclips, and/or audio clips captured by the target operator's mobiledevice responsive to manual triggers during the digital draft procedure;a timeseries of locations and orientations of the target operator'smobile device (e.g., within the facility globally or relative to themachine specifically, such as calculated by the mobile device or remotecomputer system as described below) throughout completion of this firstinstance of the digital draft procedure; times and/or locations of themobile device at which the target operator triggered the mobile deviceto record these images, video clips, and/or audio clips during steps ofthe digital draft procedure; and/or times or locations of the mobiledevice at which the target operator manually indicated at the mobiledevice that she viewed a display specified for manual data recordationin the digital draft procedure. The remote computer system can then linkeach of these images, video clips, audio clips, and/or manual datarecordation flags to a corresponding step and input field in theprocedure and to a corresponding location and orientation (or range oflocations and orientations) of the mobile device (e.g., in the facilitygenerally or relative to the machine more specifically).

For a first step in the procedure containing an input field, the remotecomputer system can then: retrieve an image, video clip, audio clip,and/or manual data recordation flag recorded during this step of thefirst instance of the digital draft procedure; retrieve a location(s)and orientation(s) of the target operator's mobile device during thisstep of the first instance of the digital draft procedure; serve thesedata to the administrator via the administrator portal; and prompt theadministrator to manually indicate a display depicted in these data andto specify a type of the display. For example, the administrator portalcan render a color image or a (2D or 3D) video clip recorded by thetarget operator's mobile device during this first instance of thedigital draft procedure. The administrator can then: draw a bounding box(or bounding circle) around a display depicted in this image or videoclip (or otherwise indicate the display associated with the first stepin the image or video clip); and specify a type of the display (e.g.,analog or digital rotary dial; multi-color digital display;multi-segment digital or vane display; a split-flat display).Accordingly, the administrator portal (or the remote computer system)can: implement triangulation techniques to calculate a particularposition of the display in the facility based on the location of thebounding box in the image or video clip and the location and orientationof the target operator's mobile device when this image or video clip wasrecorded; calculate a range of positions and orientations of the mobiledevice in which this bounding box—and therefore the display—may fall inthe field of view of the camera in the mobile device based on theparticular position of the display in the facility and opticalcharacteristics of the camera; and define a spatial trigger forautomatically capturing an image or video clip during this step infuture instances of the procedure based on this range of positions andorientations. The remote computer system can also retrieve a pointer,identifier, or link to a data extraction model configured to extract avalue from an image or video of a display of a type indicated by theadministrator. The remote computer system can thus: define a spatialtrigger for capturing data from the display on the machine during thisstep of the procedure based on the position of the target operator'smobile device during completion of this step in this first instance ofthe digital draft procedure; and link this spatial trigger and a dataextraction model for extracting data from an image of this display tothis first step in the procedure.

Alternatively, the remote computer system can: interface with the targetoperator in real-time during the first instance of the digital draftprocedure to record the location and type of a display on a machinedepicted in an image or in a live video feed recorded by the targetoperator's mobile device during this digital draft procedure; and thenimplement the foregoing methods and techniques to define a spatialtrigger for capturing data from this display on the machine during thisstep of the procedure and to link this spatial trigger and a dataextraction model to this step in the procedure.

Alternatively, the remote computer system can implement artificialintelligence, computer vision, and/or other techniques to automatically:detect a display in an image or video clip recorded concurrently withmanual data entry by the target operator during a step in this firstinstance of the digital draft procedure; and identify a type of thisdisplay. The remote computer system can then implement the foregoingmethods and techniques to define a spatial trigger for capturing datafrom this display on the machine during this step of the procedure andto link this spatial trigger and a data extraction model to this step inthe procedure.

The remote computer system can execute the foregoing methods andtechniques to generate a spatial trigger and link to a data extractionmodel for each step of the procedure associated with an input field. Theremote computer system can then compile these spatial triggers and thedigital draft procedure to generate an augmented digital procedure, asdescribed in U.S. Patent Application No. APPR-M03-US. In particular, theremote computer system can compile the set of descriptions of steps inthe procedure, spatial triggers for capturing data from displays on (ornear) the machine during steps of the procedure, definitions for typesor formats of data captured during these steps, and links between steps,spatial triggers, capture types, and input fields for these steps intoan augmented digital procedure executable on a handheld computing device(e.g., a smartphone, a tablet) or on an augmented reality headset toguide operators through instances of this procedure and to automaticallycapture and store data on behalf of these operators during theseinstances of the procedure.

Thus, in this implementation, the remote computer system can later:serve an instance of this augmented digital procedure to a mobile devicecarried by or assigned to an operator in preparation for performing thisprocedure at the same or similar machine; select an image captured by anoptical sensor in this mobile device when the location of this mobiledevice falls within a threshold linear and angular offset from thespatial trigger defined in a current step of the augmented digitalprocedure; extract a value from a region of the image depicting adisplay according to a capture type and a data extraction modelassociated with this step in the augmented digital procedure; and thenautomatically store this value—linked to an input field associated withthis step of the augmented digital procedure—in a procedure file forthis instance of the augmented digital procedure.

However, the remote computer system, the administrator portal, and/or atarget operator's mobile device can implement any other method ortechnique to define an automatic capture trigger for a step of theaugmented digital procedure, such as automatically or in cooperationwith the administrator.

4.7 Data Extraction Models

As described above, the remote computer system can associate an inputfield in a step in the procedure with a data extraction model, such asselected from a set or database of predefined data extraction modelsconfigured to extract values from various types of displays, as shown inFIGS. 4B, 5A, 5B, and 5C.

In one implementation, a data extraction model can define an artificialintelligence or computer vision model trained on images of a particulartype of display (e.g., an analog rotary dial; a 7-segment display) todetect a display of this particular type in an image, to extractfeatures representative of a value represented on the display, tointerpret this value based on these images, and to format this valueaccording to particular capture type. In this implementation, dataextraction models for various display types—such as an analog rotarydial, a digital rotary dial, a multi-color digital display, amulti-segment digital display, a vane display, and/or a split-flatdisplay—can be trained with sets of labeled training images and can bestored in a database of data extraction models. For example, the dataextraction model database can store: a dial-type data extraction modeldefining detection of a dial in an image, detection of a needle withinthe dial in the image, and interpretation of a value from the imagebased on the dial and the needle; a multi-segment-type data extractionmodel defining detection of a multi-segment display in an image andinterpretation of a value from the multi-segment display; and/or adigital-type data extraction model defining detection of a digitaldisplay in an image and interpretation of a value from the digitaldisplay.

The remote computer system can implement methods and techniquesdescribed above to link a particular data extraction model in thisdatabase to an input field in a step of a procedure during generation ofan augmented digital procedure and can later retrieve a data extractionmodel—from the database—corresponding to a particular step of theprocedure in order to automatically interpret a value from an imagecaptured during completion of this step in an instance of this augmenteddigital procedure, as described below.

5. Mobile Device Localization

As an operator wearing or carrying a mobile device walks through thefacility and approaches a machine in preparation for performing aninstance of the augmented digital procedure scheduled for or associatedwith this machine, the mobile device (and/or the remote computer system)can track its location within the facility and identify a particularmachine with which the operator is interfacing based on this location,as shown in FIGS. 3 and 7. For example, the mobile device can: determinethat the mobile device is occupying a particular campus based on themobile device's current geospatial (e.g., GPS) coordinates; determinethe building, floor, and/or room that the mobile device is occupyingbased on wireless connectivity in the space occupied by the mobiledevice; and then compare features detected in images recorded by acamera in the mobile device to a 2D or 3D localization map of thebuilding, floor, and/or room in the facility in order to determine theposition and orientation of the mobile device in real space. In thisexample, the mobile device (or a remote computer system) can then querya map of machines throughout the facility for a particular machineadjacent and/or facing the mobile device—and therefore near theoperator—based on the position and orientation of the mobile device inreal space. Alternatively, the mobile device can identify the particularmachine directly by matching a constellation of features detected inimages recorded by the camera to a known, unique constellation offeatures associated with this particular machine.

The mobile device can regularly execute this process to monitor itsposition and orientation within the facility and detect machines nearby.

Alternatively, the mobile device can stream data to the remote computersystem—such as via a computer network—while in operation within thefacility, and the remote computer system can implement similar methodsand techniques to detect and track the location of the mobile device in(near) real-time based on these data as the mobile device movesthroughout the facility.

6. Procedure Selection

In one implementation shown in FIGS. 1 and 3, when the operator stopsfor more than a threshold duration of time (e.g., ten seconds) or whenthe mobile device determines that its location has moved less than athreshold distance within a period of time (e.g., less than one meter ina ten-second period), the mobile device can: query the digital proceduredatabase for an augmented digital procedure associated with a machinenearest the current position of the mobile device; and automaticallyload an instance of this augmented digital procedure for this machine,such as if this augmented digital procedure is scheduled for completionwithin a current time window. In another implementation, the mobiledevice can implement multiple rules to trigger a new instance of theaugmented digital procedure (e.g., in order to suppress falsepositives), such as when the operator pauses in one location for morethan a threshold duration of time (e.g., remains within a one-meterfloor area within a ten-second period) and continues to look in thedirection of a particular machine scheduled for an instance of theaugmented digital procedure (e.g., with her head oriented toward themachine for at least five seconds).

In a similar implementation, the mobile device can: rank machines in thefacility by proximity to the current location of the mobile device;render a list of these machines ordered by their rank on a display ofthe mobile device; prompt the operator to select from the list; anddownload an instance of a particular augmented digital procedureassociated with a machine selected by the operator.

Alternatively, the operator can manually select the particular machinedirectly from a dropdown list of machines, select the particular machinefrom a map of the facility, or select the particular augmented digitalprocedure directly from a dropdown list of procedures. However, themobile device can implement any other method or technique to select andload an instance of an augmented digital procedure.

7. User-Driven Data Capture

Once the augmented digital procedure is loaded onto the mobile device,the mobile device can present content in this augmented digitalprocedure to the operator, as shown in FIG. 1. For example, the mobiledevice can: render each discrete page of the procedure document insequential order; or consecutively render individual steps (or clustersof steps) outlined in the augmented digital procedure as the operatorcompletes these steps over time.

As the operator performs a step of the augmented digital procedure atthe mobile device, the operator may select a data input region adjacentthis step in order to manually trigger data capture by the mobiledevice. For example, when the operator selects an input region adjacenta step tagged for manual numerical input, the mobile device can: rendera 10-digit keypad; record, geotag, and timestamp a numerical valueentered by the operator into this input region via the keypad; and storethis numerical value—associated with this step—in the procedure file forthis instance of the augmented digital procedure. In a similar example,when the operator selects an input region adjacent a step tagged formanual text string input, the mobile device can: render a keyboard;record, geotag, and timestamp a text string entered by the operator intothis input region via the keyboard; and store this textstring—associated with this step—in the procedure file for this instanceof the augmented digital procedure. When the operator selects a virtual“record” button over or adjacent a step tagged for voice capture, themobile device can: record, geotag, and timestamp an audio file while therecord button is depressed or between two consecutive selections of thevirtual record button; and store this audio file—associated with thisstep—in the procedure file for this instance of the augmented digitalprocedure.

Furthermore, when the operator selects a virtual “record” button over oradjacent a step tagged for image capture, the mobile device can: record,geotag, and timestamp an image (or a video, such as two-seconds induration); and store this image (or this video)—associated with thisstep—in the procedure file for this instance of the augmented digitalprocedure. If this step in the augmented digital procedure furtherincludes a definition for automatic data extraction, the mobile device(or the remote computer system) can implement methods and techniquesdescribed below to: select a particular image in this video, such as aparticular image characterized by minimal blur; detect a display in aregion of this image; extract a value shown on the display, etc.depicted in this region of the image; and store this value in theprocedure file for this instance of the augmented digital procedure,such as in association with this step and the time that the image wasrecorded by the mobile device.

8. Guided Image Capture

In another implementation shown in FIGS. 1 and 2, the mobile deviceincludes an augmented reality headset with a heads-up display. In thisimplementation, as an operator wearing this mobile device moves throughthe facility and stops at a particular machine, the mobile device can:track its position and orientation within the facility; identify anaugmented digital procedure relevant to this particular machine; andautomatically load an instance of this augmented digital procedure. Asthe operator opens a step—in this augmented digital procedure—specifyingautomatic data capture from a particular display on the machine, themobile device can: calculate the global position of the display relativeto the mobile device; render a description of the step on the heads-updisplay; and render a prompt to face a display associated with thisstep, such as by prompting the operator to turn left, turn right, and/orwalk forward toward a known location of this display. During this step,the camera can regularly record images (e.g., at a rate of 24 Hz), andthe mobile device can process these images to both localize the mobiledevice within the facility and to detect a known optical fiducial nearthe display or to detect the display directly (e.g., if a model of thedisplay is linked to this step) in these images. When the mobile devicethus detects the display, the mobile device can project a staticpointer, animation, or other indicator on the heads-up display at aposition that locates this indicator over the display in the operator'sfield of view, thereby highlighting the display for the operator. Themobile device can then: confirm whether the camera is facing thedisplay; confirm whether the mobile device is within a maximum distanceof the display; confirm whether the mobile device is sufficiently static(e.g., to minimize blur); and, accordingly, store a last image, a nextimage, or a current “burst” of images recorded by the camera. The mobiledevice can then either: execute methods and techniques described belowto extract a value presented on the display from this image inreal-time; or return the image to the remote computer system forextraction of this value from this image.

Similarly, as the operator opens a step—in the augmented digitalprocedure—specifying data capture from support equipment labeled with abarcode or QR code, the mobile device can scan images recorded by thecamera for this barcode or QR code. Upon detecting this barcode or QRcode in a last image captured by the camera, the mobile device canhighlight this support equipment on the heads-up display and store alast image, a next image, or a current “burst” of images captured by thecamera. The mobile device can also render prompts on the heads-updisplay: to move the mobile device nearer to the display; to center thedisplay in the operator's field of view or the field of view of thecamera in the mobile device; or to remain still while the mobile devicecaptures images of this display.

In another implementation in which the mobile device is a handheldcomputing device (e.g., a smartphone, a tablet), the mobile device canimplement similar methods and techniques to augment a live video feedcaptured by a camera in the mobile device with guidance related tocapturing an image or video clip of a display on the machine. Forexample, during a step of the augmented digital procedure, the mobiledevice can: render a video feed captured by the optical sensor in themobile device on a viewfinder (e.g., an internal display) of the mobiledevice; access a virtual guide associated with this step of theaugmented digital procedure; render the virtual guide over the videofeed on the viewfinder; prompt the operator to align the virtualguide—rendered on the viewfinder—with a display on the machineassociated with this step of the augmented digital procedure; record animage through the optical sensor in response to detecting alignmentbetween the virtual guide rendered on the viewfinder and the display onthe machine; and then flag this image for data extraction according tothe input field associated with this step of the augmented digitalprocedure. In this implementation, the mobile device (or the remotecomputer system) can also implement methods and techniques describedabove to track the location of the mobile device in the facility,estimate the position of the field of view of the optical sensorrelative to the display, and render augmented guidance on the viewfinderon the internal display of the mobile device to direct the operator toalign the optical sensor to the display on the machine.

Furthermore, in the foregoing implementations, the mobile device can:prompt the operator to orient the mobile device toward the display onthe machine; render virtual guidance to guide the operator in achievingproper orientation of the mobile device relative to the display; andimplement methods and techniques described above and below toautomatically record an image of the display on the machine when thedisplay is predicted to fall in the field of view of the optical sensorbased on a known location of the display in the facility (such as storedin a localization map of the facility), the detected location of themobile device, and optical characteristics of the optical sensor in themobile device.

However, the mobile device can implement any other methods or techniquesto guide the operator when recording an image or video clip of a displayon the machine (or an a supporting equipment unit nearby).

9. Passive Image Capture

The mobile device can additionally or alternatively record an image orvideo clip automatically (or “passively”) while the operator performs astep in this instance of the augmented digital procedure.

In one implementation, once the mobile device automatically selects,loads, and initiates an instance of the augmented digital procedureand/or once the operator selects or confirms this instance of theaugmented digital procedure, the mobile device can: initiate capture ofa video through the camera in the mobile device; and timestamp andgeotag each frame in this video feed or record a synchronized timeseriesof locations of the mobile device (e.g., within the facility generallyor relative to the machine more specifically). For example, the mobiledevice can annotate a frame of the video with: a GPS location of mobiledevice; a facility associated with this GPS location; a set of wirelessnetworks connected to the mobile device; a room or floor in the facilityassociated with this set of wireless networks; and/or a specificlocation and orientation of the mobile device—determined by the mobiledevice by comparing features in the frame to the localization map—at thetime the frame was recorded by the mobile device.

In this implementation, once the operator completes the last step in theprocedure, terminates the procedure, or moves away from the machine, themobile device can cease capture of this video. The mobile device canthen process frames in this video locally according to methods andtechniques described below or upload this video to the remote computersystem for such processing.

For example, the mobile device can: record a continuous video (e.g., ata frame rate of 24 Hz) during completion of this procedure by theoperator; record a synchronized timeseries of locations of the mobiledevice; and record timestamps or a timeseries of start and stop times ofsteps and manual capture times in this instance of the augmented digitalprocedure throughout the augmented digital procedure. Upon completion ofthis instance of the augmented digital procedure, the mobile device canupload these data to a remote database, and the remote computer systemcan process these data as described below to extract values from framesin the video feed and to write these values to a procedure file for thisinstance of the augmented digital procedure. In particular, the remotecomputer system can process geotags linked or synchronized to frames inthis video to isolate a subset of these frames that were recorded by thecamera when one or more displays of interest—designated by the augmenteddigital procedure—fell within the field of view of the camera and fellwithin a maximum distance of the mobile device. The remote computersystem can then implement methods and techniques described below toextract values presented on these displays, during this instance of theaugmented digital procedure, from this subset of images.

Alternately, the mobile device can stream these data to the remotecomputer system in (near) real-time during completion of this instanceof the augmented digital procedure, and the remote computer system canexecute methods described herein to extract values from these data andto write these values to a procedure file for the augmented digitalprocedure accordingly.

Yet alternatively, the mobile device can execute these methods locallyin (near) real-time to extract values from these data recorded by themobile device during this instance of the augmented digital procedureand to populate a procedure file for this augmented digital procedurewith these values. For example, during operation, the mobile device cancapture a video feed via its integrated optical sensor. Once a nextframe in the video feed is captured and stored in a local cache, themobile device can extract a constellation of features from the image andcalculate a location and orientation of the mobile device—within a localcopy of a localization map of the facility—that aligns thisconstellation of features to corresponding features in the localizationmap. The mobile device can then: project a field of view of the opticalsensor into the localization map based on the first location of themobile device, the first orientation of the mobile device, and knownoptical characteristics of the optical sensor; and predict presence ofthe display in the field of view of the optical sensor in response tothe field of view of the optical sensor—projected onto the localizationmap—containing a known or stored location of the display represented inthe localization map.

A handheld computing device—such as a smartphone, tablet, or othermobile computer—can implement methods and techniques similar to thosedescribed above to present augmented reality guidance to an operatorperforming an instance of the augmented digital procedure, to capturedata when triggered manually by the operator, to selectively capturedata automatically, and/or to capture images continuously—for real-timeor asynchronous processing and data extraction—during this instance ofthe augmented digital procedure.

(Furthermore, a second mobile device carried or worn by a secondaryoperator working in proximity to the machine—as the (primary) operatorperforms steps of this augmented digital procedure—can implement similarmethods and techniques to passively capture data representation a stateof the machine or of a process occurring at the machine. This secondmobile device or the remote computer system can then implement similarmethods and techniques to extract values from these data captured by thesecond mobile device and to write these data to the procedure file forthe augmented digital procedure performed by the (primary) operator.

10. Image Filtering

Therefore, the mobile device can collect single images, a sequence (or“burst”) of images, a video clip, or a continuous video duringcompletion of a step of the augmented digital procedure by the operator.The mobile device can then filter these images or video frames toisolate one or a subset of images or frames from which to extract valuespresented on the display. For example, for a step in the augmenteddigital procedure designating automatic data capture from a particulardisplay on a machine of a particular make, model, configuration, and/orclass, the mobile device (or the remote computer system) can isolate oneor a small set of images: recorded by the mobile device while the stepwas active; recorded when the mobile device was nearest the known orpredicted location of the display; recorded when the orientation of themobile device located the known or predicted location of the displaynear the center of the field of view of the camera; and/or recordedwhile the mobile device experienced minimal rotation about any axis andminimal horizontal and vertical translation relative to the primary axisof the camera.

The mobile device (or the more computer system) can then flag this imageor subset of images for data extraction, as described below.

11. Data Extraction

The mobile device (or the remote computer system) can thus select animage captured by the optical sensor in the mobile device in Block S130and extract a value from a region of the image depicting the display inBlock S142.

11.1 Display Detection

In one implementation, the mobile device (or the remote computer system)first scans the image for a region depicting the display. For example,the mobile device can implement methods and techniques described aboveto estimate a location of the display in the field of view of theoptical sensor in the mobile device based on a known location of thedisplay (e.g., as represented in a localization map of the facility) andthe location and orientation of the mobile device at the time the imagewas captured. In this example, the mobile device can then isolate aparticular region in the image predicted to depict the display based onthis estimated location of the display in the field of view of theoptical sensor at the time the image was recorded. The mobile device canthen access a display model—associated with the concurrent step of theaugmented digital procedure—depicting or representing identifyingfeatures of the display, content rendered on the display, or knownfeatures near the display. For example, the display model can include atemplate image of the display, a sparse “fingerprint” image of thedisplay, or a parametric model representing color, spatial, and/orgeometric features representative of the display and defining a boundaryof the display. The mobile device can then scan the particular region ofthe image (and an extended area of the image around this particularregion) for features representative of the display based on the displaymodel. For example, the mobile device can: map features in (and near)the particular region of the image to like features represented in thedisplay model; and then calculate a transform that aligns these featuresin the display model to the corresponding features in the particularregion of the image. The mobile device can thus project the boundary ofthe display—defined in the display model—onto the image. The mobiledevice can also: crop, extract, or isolate a segment of the imagecontained within this projected boundary; and apply the inverse of theforegoing transform to this segment of the image in order to “flatten”or “unwarp” this depiction of the display.

Alternatively, the mobile device (or the remote computer system) canisolate a segment or region of the image depicting the display directlybased on the known location of the display in the facility and locationand orientation of the mobile device at the time the image was recorded.

Yet alternatively, the mobile device (or the remote computer system)can: scan an image—recorded during a particular step of the augmenteddigital procedure—for features representative of the display, such asbased on a display model associated with this step of the augmenteddigital procedure as described above; and then extract a segment orregion of the image in which the display is detected responsive toidentifying a high-confidence match between features in the image andfeatures represented in the display model.

However, the mobile device (or the remote computer system) can implementany other method or technique to detect a display in an image and toisolate or extract a region or segment of an image depicting a display.

11.2 Data Extraction Model

As shown in FIGS. 2, 3, 4B, 5A, 5B, and 5C, one variation of the methodrecites retrieving a data extraction model, from a set of dataextraction models, linked to the display and the machine in Block S162.Generally, in this variation, to extract a value presented on thedisplay from this image captured by the mobile device during thisinstance of the augmented digital procedure, the mobile device (or theremote computer system) can retrieve a data extraction model for thisstep, machine, and/or display, such as: a generic data extraction modelfor a known type of display, as specified in the augmented digitalprocedure; or a custom data extraction model constructed specificallyfor this display or for displays found in this make, model,configuration, and/or class of machines or support equipment. The mobiledevice (or the remote computer system) can then execute a sequence ofimage-processing actions to detect the display in a region of thisimage, to detect values in this region of the image, and to extractthese values from the image.

As described above, the augmented digital procedure can identify aparticular machine or equipment type, and a step in this augmenteddigital procedure associated with an input field and capture type can belinked to a data extraction model that defines extraction of values froma type of display on this machine or equipment type and specified inthis step. Thus, the mobile device can retrieve a particular dataextraction model—from a set of data extraction models stored in adatabase—specified in the augmented digital procedure, linked to thestep, and corresponding to a type of the display on the machine anddesignated for data capture by this step. The mobile device can thenextract a set of features from an isolated region of the image depictingthe display and interpret a value from this set of features according tothe particular data extraction model.

Alternatively, if the step of the augmented digital procedure is notassociated with a data extraction model (or if the image was capturedpassively by the mobile device without a link to a particular step in anaugmented digital procedure), the mobile device can identify the machinedepicted in the image, such as based on a barcode, QR code, or otheroptical fiducial on the machine detected in the image or based on aknown location of the machine in the facility and the location andorientation of the mobile device at the time the image was recorded. Inthis implementation, the mobile device can then identify a type of adisplay in the image, such as: by implementing artificial intelligenceor computer vision techniques to detect the display directly; bydetecting a particular optical fiducial associated with the display inthe image; or based on a known display type arranged on the machine andthe position of the mobile device relative to the known location of thisdisplay on the machine at the time the image was captured. The mobiledevice can then retrieve a particular data extraction model—from thedatabase of data extraction models—associated with this display type andthen implement this data extraction model to extract a value from aregion of the image depicting the display, as described below.

However, the mobile device (or the remote computer system) can implementany other method or technique to selectively retrieve a data extractionmodel defining extraction of a relevant value from an image of aparticular type of display in Block S162.

11.2.1 Example: 7-Segment Display

In one example in which the display includes a 7-segment display, themobile device (or the remote computer system) can: convert the image ofthe display to grayscale; apply Gaussian blurring (e.g., with a 5×5kernel) to reduce high-frequency noise in the grayscale image; computean edge map of the grayscale image (e.g., with a Canny edge detector);and implement a mask or model to identify a set of contours thatrepresent the boundary of the display (e.g., a rectangular area of aparticular known aspect ratio) in the edge map. The mobile device canalso project a predicted location of the display into the edge map basedon a known location of the display in the facility, a position andorientation of the mobile device at the time the image was recorded,properties of the camera, and the 3D localization map of the facilityand verify that this set of contours represent the boundary of thedisplay accordingly. The mobile device can then: extract a region of thegrayscale image bounded by this set of contours; apply a four-pointperspective transform to this region of the grayscale image to generatea warped image of the display; implement thresholding techniques toconvert the warped image to a black-and-white image; apply morphologicaloperations to reduce noise in the black-and-white image; detect digitcontours in the black-and-white image; define bounding boxes over digitcontours in the black-and-white image; and then project individualsegment regions into each bounding box. Within each bounding box, themobile device can: determine whether each individual segment region isactive based on whether the average color value of pixels within thisindividual segment region is greater or less than a threshold value; andinterpret a digit value within the bounding box based on states ofindividual segment regions contained within the bounding box. The mobiledevice can then assemble digit values from this set of bounding boxesinto a final numerical value and store this final numerical value in aprocedure file for this instance of the augmented digital procedure,such as in a particular input field corresponding to this display andthis step in the augmented digital procedure.

11.2.2 Example: Rotary Dial

In another example in which the display includes an analog or digitalrotary dial, as shown in FIGS. 1 and 2, the mobile device can: convertan image of the display to grayscale; apply Gaussian blurring to reducehigh-frequency noise in the grayscale image; compute an edge map of thegrayscale image; and implement a mask or model to identify a contourthat represents the boundary of the dial (e.g., a circular areacharacterized by a dimension relative to an adjacent rectangular orlinear contour or relative to a known optical fiducial near the dial);extract a region of the grayscale image bounded by this contour. Themobile device can also retrieve a first color of a needle (e.g., red), asecond color on the dial (e.g., white), and a third color of tick markson the dial (e.g., black) from the augmented digital procedure. Todetect the needle on the dial, the mobile device can: extract a regionof the grayscale image bounded by the contour; apply a four-pointperspective transform to this region to generate a warped color image ofthe display; implement thresholding techniques to convert the warpedimage to a black-and-red image; apply morphological operations to reducenoise in the black-and-red image; detect contours in the black-and-redimage; isolate a needle contour in the black-and-red image; andimplement line fitting techniques to locate a long axis of the needlecontour in the black-and-red image. To detect tick marks on the dial,the mobile device can: extract a region of the grayscale image boundedby the contour; apply a four-point perspective transform to this regionto generate a warped color image of the display; implement thresholdingtechniques to convert the warped image to a black-and-white image; anddetect tick contours around the edge of the black-and-white image. Themobile device can then: combine tick contours detected in theblack-and-white image and the needle contour detected in theblack-and-red image to generate a composite image; extend the tickcontours linearly toward a center of the composite image; detectintersections of the extended tick contours; locate an origin of a polarcoordinate system at the centroid of the these extended tick contourintersections; locate a polar axis of the polar coordinate systemextending from the origin to a lower-most tick contained on a left sideof the composite image (or at a tick contour at a location correspondingto a lowest registered value on the dial, as indicated by a model of thedial or by an optical fiducial detected in the original image); andcalculate an angle between the polar axis and the long axis of theneedle contour. Finally, based on a known register range of the dial orbased on a predefined model for converting needle angle to registeredvalue for this dial, the mobile device can convert the angle of theneedle contour into a register value and store this register value inthe procedure file for this instance of the augmented digital procedure,such as in a particular input field corresponding to this display andthis step in the augmented digital procedure.

In a similar example in which the display includes an analog dial, themobile device (or the remote computer system) can retrieve a dataextraction model defining detection of a dial in an image, detection ofa needle within the dial in the image, and interpretation of a valuefrom the image based on the position of the needle relative to the dial.In this example, the mobile device can also retrieve a dial mask—linkedto the step of the augmented digital procedure—containing a set ofvirtual dial indicators associated with the analog dial on the machineand configured to indicate values represented by a needle on the dialwhen the dial mask is projected onto a region of an image depicting theanalog dial on the machine (or a dial of this type and scale on the sameor other machine). Then, in order to extract a value from a region of animage depicting this analog dial, the mobile device can: implementmethods and techniques described above to detect the analog dial in aregion of an image; project the dial mask onto the analog dial thusdetected in the image—such as based on an estimated position of themobile device relative to the analog dial at the time the image wascaptured at the mobile device or by calculating a transform that alignsthe dial mask to features detected on or near the analog dial in thisregion of the image—and thus combine the dial mask with the region ofthe image to form a composite image of the analog dial annotated withdial indicator values. (Alternatively, the mobile device can extractthis region of the image depicting the analog dial, dewarp thisextracted region of the image, and align the dial mask over the dewarpedregion of the image to form a composite image annotated with dialindicator values.) The mobile device can then: detect an analog needlein this composite image, such as by scanning the composite image for aline of pixels in a known color of the needle in this analog dial asindicated in the data extraction model, linked to the step in theaugmented digital procedure, or linked to the machine more generally.Finally, the mobile device can interpret a value presented on the analogdial at the time the image was captured based on a position of theanalog needle relative to the set of virtual dial indicators annotatedover the composite image.

11.2.3 Example: Digital Color Display

In yet another example in which the display includes a digital colordisplay, as shown in FIGS. 1 and 5C, the mobile device can extract textrendered on this display by: converting an image of the display tograyscale; applying Gaussian blurring to reduce high-frequency noise inthe grayscale image; computing an edge map of the grayscale image;implementing a mask or model to identify a set of contours thatrepresent the boundary of the display (e.g., a rectangular area of aparticular known aspect ratio; a keystone geometry as a function ofviewing angle); extracting a region of the image bounded by this set ofcontours; applying a four-point perspective transform to this region togenerate a warped image of the display; implementing thresholdingtechniques to convert the warped image to a black-and-white image;applying morphological operations to reduce noise in the black-and-whiteimage; and then implementing optical character recognition techniques todetect and read text in this de-noised black-and-white image. The mobiledevice can then store this text in the procedure file for this instanceof the augmented digital procedure.

12. Timeseries Data

The mobile device (or the remote computer system) can: execute theforegoing process for multiple images of one display recorded during onestep of the augmented digital procedure; associate a value extractedfrom an image in this set with a time that the image was recorded by themobile device; and thus generate a timeseries of values presented on thedisplay during completion of this step in this instance of the augmenteddigital procedure.

For example, during a step of the augmented digital procedure, themobile device can: select a sequence of images captured by the opticalsensor over a duration of time specified in the step of the augmenteddigital procedure; implement the foregoing methods and techniques toextract a sequence of values from regions of images in this sequencethat depict the display designated for timeseries data capture by thisstep of the augmented digital procedure; associate each value in thesequence with a time that the corresponding image was captured by themobile device; aggregate the sequence of timestamped values into atimeseries of values displayed by the display during this step; and thenstore this timeseries of values—linked to the input field associatedwith this step—in the procedure file for this instance of the augmenteddigital procedure.

Additionally or alternatively, the mobile device (or the remote computersystem) can: filter this timeseries of values in order to rejectoutliers and write this filtered timeseries of values to the procedurefile; or calculate a mean, median, or mode of this timeseries of valuesand store this calculated value in the procedure file for this instanceof the augmented digital procedure.

In another implementation in which data presented by the display settlesover a period of time during the step of the augmented digitalprocedure, the mobile device can: detect bounce or variance in thetimeseries of values extracted from this sequence of images; detect asegment of this timeseries depicting stable or “settled” values;calculate a mean, median, or mode of this segment of the timeseries; andstore this calculated value in the procedure file for this instance ofthe augmented digital procedure. For example, the mobile device can:select a sequence of images captured by the optical sensor over aduration of time during a step of the augmented digital procedure;extract a sequence of values from regions in the sequence of imagesdepicting the display, such as described above; calculate rates ofchange across values in this sequence of values; and store a particularvalue—in this sequence—following a segment of this sequence thatexhibits less than a threshold rate of change. Furthermore, in thisimplementation, the mobile device (or the remote computer system) canexecute the foregoing process in real-time during capture of theseimages during this step of the augmented digital procedure and canselectively deactivate capture and/or extraction of values from theseimages upon detecting a stable or “settled” value on this display.

13. Display Cycle

In one variation, the mobile device can also distinguish differentcycles or frames of data rendered on the display during capture ofimages by the mobile device during a step of the augmented digitalprocedure.

In one implementation in which the display cycles through presentationof data of different types, such as specified in the augmented digitalprocedure, the mobile device (or the remote computer system) can:implement methods and techniques described above to transform a sequenceof images reordered by the mobile device during a step of the augmenteddigital procedure into a timeseries of values; group similar consecutivevalues in this timeseries of values; link each group of values to onecycle or frame of the display, such as based on a range of valuesassociated with each cycle or frame; select a mean, media, mode, orother value from each group of similar consecutive values; and storethese values in the procedure file with links to corresponding cycles orframes rendered by the display.

In another implementation, the mobile device can: capture a sequence ofimages during a step of the augmented digital procedure. The mobiledevice (or the remote computer system) can then: detect the display in afirst region of a first image in this sequence; detect a first featurein the first region of the first image corresponding to a first frametype rendered on the display, such as defined in the step of theaugmented digital procedure or associated with the machine moregenerally; retrieve a first data extraction model associated with thisfirst feature—and therefore with this first frame type—in this step ofthe augmented digital procedure; implement methods and techniquesdescribed above to extract a first value from the first region of thefirst image according to this first data extraction model; and thenstore this first value—in a first format associated with this firstframe type in the step—in a procedure file for this augmented digitalprocedure.

In this implementation, the mobile device (or the remote computersystem) can also: detect the display in a second region of a secondimage in this sequence; and detect a second feature in the second regionof the second image corresponding to a second frame type rendered on thedisplay, such as defined in the step of the augmented digital procedureor associated with the machine more generally. If this step of theaugmented digital procedure does not specify recordation of a value fromthe second frame type or if this step of the augmented digital procedurespecifically specifies no recordation of a value from the second frametype, the mobile device can discard the second image. Alternatively, ifthis step of the augmented digital procedure does specify recordation ofa value from the second frame type, the mobile device can: retrieve asecond data extraction model associated with this second feature—andtherefore with this second frame type—in this step of the augmenteddigital procedure; implement methods and techniques described above toextract a second value from the second region of the second imageaccording to this second data extraction model; and then store thissecond value—in a second format associated with the second frame type inthe step—in the procedure file for this augmented digital procedure.

13. Data Calibration

In one variation, the remote computer system (or the mobile device) canalso verify data extracted from an image by confirming that a displaydetected in the image was calibrated (e.g., met calibration limitationsassigned to the machine) at the time the image was recorded. Forexample, the computer system can: access calibration records forequipment in the facility, such as a database or table listing dates andtimes that individual machines and supporting equipment in the facilitywere last calibrated; and verify that a duration of time from thetimestamp on the image to a last calibration time for the machine isless than a specified calibration duration for the machine. If so, theremote computer system can verify this value extracted from this imageand store this value accordingly. If not, the remote computer system candetermine that the machine was not calibrated at the time of this datacapture and either discard this value or store this value with a flag asnot verified.

14. Data Verification

In one variation shown in FIG. 2, the mobile device and/or the remotecomputer system can: execute the foregoing methods and techniques topassively capture an image of a machine or display during a step of anaugmented digital procedure and to automatically extract a value from aregion of this image depicting a display designated for data captureduring the concurrent step of the augmented digital procedure; and thencompare this automatically-extracted value to a value manually recordedby the operator during this step of the augmented digital procedure inorder to verify accuracy of these data; and then selectively flag thisstep of the augmented digital procedure and these data for review (e.g.,by a supervisor)—such as in real-time or upon conclusion of theaugmented digital procedure—responsive to detecting a difference betweenthese automatically-extracted and manually-recorded values.

For example, the mobile device can: passively capture a video feedduring a step of the augmented digital procedure; select an image fromthis video feed when a display on a machine designated for data captureand verification in the step falls within the field of view of themobile device's camera; implement the foregoing methods and techniquesto extract a value from this image; and then store this value—as averification value linked to the step—in the procedure file for thisinstance of the augmented digital procedure. Furthermore, during thisstep, the mobile device can also prompt the operator to manually recorda reference value for this step according to the augmented digitalprocedure. For example, the mobile device can: render a graphical userinterface and record a reference value manually entered into thegraphical user interface by the user; or record an audio clip of theuser speaking the reference value and then implement speech-to-texttechniques to extract this reference value from the audio clip. Themobile device can then: store the reference value—linked to this step inthe augmented digital procedure—in the procedure file; compare theverification value and the reference value; and then flag the procedurefile (e.g., this step specifically or this instance of the augmenteddigital procedure more generally) for review in response to detecting adifference between the verification value and the reference value. Forexample, in response to detecting a difference between the verificationvalue and the reference value, the mobile device (or the remote computersystem) can serve these values, a video clip spanning capture of thesevalues, and a prompt to review these data to a second device—such ascarried or monitored by a supervisor, an administrator, or a secondoperator—for review, such as in real-time or following completion ofthis instance of the augmented digital procedure. Alternatively, inresponse to detecting a difference between the verification value andthe reference value, the mobile device can serve a prompt the operatorto confirm the reference value before transitioning to a next step ofthe augmented digital procedure.

15. Operator Feedback and Processing Resources

In another variation, the mobile device (or the remote computer system)can return confirmation to the operator that a value—designated forcapture during the current step of the augmented digital procedure—wassuccessfully captured and then enable or automatically transition to anext step of the augmented digital procedure.

In one implementation in which the mobile device transmits an image orvideo feed to the remote computer system for processing and dataextraction during a step of an augmented digital procedure, the remotecomputer system can: implement the foregoing method and technique toextract a value from this image or video feed; store this image in aprocedure file for this instance of the augmented digital procedure; andthen transmit confirmation of data capture to the mobile deviceaccordingly. The mobile device can then close this step of the augmenteddigital procedure and automatically open a next step of the augmenteddigital procedure in response to receipt of confirmation of data capturefrom the remote computer system.

16. Procedure File

The mobile device and/or the remote computer system can write datarecorded manually by the operator and data recorded automatically by themobile device and/or by the remote computer system to a unique procedurefile for this unique instance of the augmented digital procedure, asshown in FIGS. 1, 2, 3, and 6. For example, the mobile device can geotagand timestamp these active and passive data in the procedure file andcan write an identifier of the particular machine (e.g., a machine make,model, and serial number), an identifier of the operator (e.g., anemployee ID number), and an augmented digital procedure identifier(e.g., procedure ID and revision number) to this procedure file. Theremote computer system can then load data recoded during this instanceof the augmented digital procedure into corresponding input fields inthe procedure file and then upload the procedure file to a remotedatabase following conclusion of this instance of the augmented digitalprocedure.

17. Timing

The mobile device can locally execute the foregoing methods andtechniques in (near) real-time during completion of a step of theaugmented digital procedure and can serve feedback to the operator toconfirm that a value was detected on a display and stored, such as byrendering augmented reality content on a heads-up display of the mobiledevice. In this implementation, the mobile device can also serve aprompt to the operator in (near) real-time to verify the value derivedautomatically by the mobile device during the current step of theaugmented digital procedure; and the mobile device can write this valueto the procedure file or repeat the foregoing methods and techniques toextract a new value for the current step of the augmented digitalprocedure based on feedback provided by the operator.

Alternatively, the mobile device or the remote computer system canexecute the foregoing methods and techniques asynchronously. Forexample, the mobile device can upload image data to a remote database,such as in real-time, upon conclusion of a step of the augmented digitalprocedure, or upon conclusion of this instance of the augmented digitalprocedure. The remote computer system can then implement methods andtechniques described above to process these images, to extract valuesfrom displays depicted in these images, and to store these values in theprocedure file for this instance of the augmented digital procedure.

18. Tele-Presence

In one variation, the foregoing methods and techniques are executed byan autonomous robotic system deployed in the facility, such as anautonomous tele-presence robot including a camera and wirelesscommunication module mounted to an autonomous wheeled platform. In thisvariation, the robotic system can autonomously navigate through thefacility—such as up to displays arranged on fixed and mobile equipmentwithin the facility—and record images of these displays. The roboticsystem or the remote computer system can then extract values from theseimages according to processes described above and write these data to aremote database. An autonomous or human supervisor or administrator maythen review these data to verify operation of equipment throughout thefacility, such as without requiring human personnel to physicallyinterface with this equipment.

19. Fixed Sensor

In another variation shown in FIG. 6, a sensor block (or “fixedsensor”)—including a camera and (wireless) communication module—isfixedly mounted to a single machine or installed in a fixed position inthe facility and oriented to face one or a set of displays on one ormore machines in the facility. In this variation, the sensor block canregularly or intermittently record images of these displays, and thesensor block or the remote computer system can then extract values fromthese images, as described above, and store these data in a remotedatabase. The remote computer system can therefore interface with fixedsensors deployed on or near displays within the facility to retrievedata from these non-networked (or “siloed”) machines without requiringhuman personnel to physically interface with this equipment.

19.1 Continuous Monitoring

In one implementation, the sensor block captures a continuous video feed(e.g., at a frame rate of 24 H), an intermittent video feed (e.g., onetwo-second video clip per minute), or intermittent images (e.g., oneimage per ten-minute interval) and returns this video feed or image tothe remote computer system. The remote computer system then implementsthe foregoing methods and techniques to extract a value from this videofeed or image. For example, the computing device can retrieve a dataextraction model for this video feed or image based on: known locationsand orientations of the sensor block and other machines in the facilityand known locations of displays on these machines; a feature identifyinga particular machine detected in the video feed or image and known typeand location of a display on this machine; a feature identifying thedisplay directly in this video feed or image; a data extraction modellinked to the sensor block generally or to a particular region in afield of view of the camera in the sensor block, such as by anadministrator or technician when the sensor block was installed in thefacility.

The remote computer system can then write a timestamped value—extractedfrom a video feed or image received from the sensor block—to a machinefile associated with the machine depicted in the video feed or image.Additionally or alternatively, the remote computer system can: access aschedule of procedures performed at this machine; identify a particularprocedure occurring on the machine at a time that this video feed orimage was recorded by the sensor block; and then write this timestampedvalue—extracted from this video feed or image—to a procedure fileassociated with this particular procedure at this machine. The remotecomputer system can thus distribute values extracted from data capturedby the fixed sensor block to procedure files for procedures performed atcorresponding times at a corresponding machine in the facility.

19.2 Cooperative Fixed Sensor and Mobile Device

In another implementation shown in FIG. 5C, the remote computer systemselectively triggers the fixed sensor block to capture a video feedand/or image when a procedure is active at the machine in the field ofview of the sensor block.

For example, the remote computer system can trigger the sensor block tocapture a video feed in response to activation of an augmented digitalprocedure at a mobile device carried by an operator. The mobile deviceand/or the remote computer system can then execute the foregoing Blocksof the method to extract values from both images captured by the mobiledevice and images captured by the sensor block during completion of thisaugmented digital procedure. The remote computer system can then thesestore values in the procedure file for this augmented digital procedure.

Alternatively, the remote computer system (or the mobile device) can:default to writing values extracted from images captured by the mobiledevice to the procedure file for this step of the augmented digitalprocedure; and implement the foregoing methods and techniques to trackthe location of the mobile device during a step of the augmented digitalprocedure to estimate (or detect) the position of a display on themachine—designated for image capture during this step of the augmenteddigital procedure—in the field of view of the sensor block. Then, if theestimated (or detected) position of the display falls outside of thefield of view of the mobile device during a particular time during thisstep of the augmented digital procedure, the remote computer system can:access an image captured by the sensor block at approximately thisparticular time; and extract a value, presented on the display atapproximately this particular time, from a region of this image; andstore this value—associated with the particular time—in the procedurefile in place of a value extracted from a concurrent image captured bythe mobile device (which may not depict the display on the machine).

Similarly, in this implementation, the remote computer system can writea value extracted from a first image recorded by the fixed sensorblock—in place of a value extracted from a concurrent image captured bythe mobile device during this instance of the augmented digitalprocedure—if the concurrent image captured by the mobile device isblurry, if glare is detected in a region of the concurrent imagedepicting the display, or if this concurrent image is otherwisedefective or of low quality; and vice versa.

(Furthermore, in this variation, the fixed sensor block can execute theforegoing processes locally to extract value from images captured by thesensor block and then return these values to the remote computer systemfor recordation.)

19.3 Autonomous Mobile Camera

In another implementation shown in FIG. 6, the remote computer systemcan implement similar methods and techniques to extract data—rendered ona display of a machine in the facility—from an image captured by anautonomous mobile camera deployed within the facility and to selectivelystore these extracted data in procedure files and/or other filesassociated with this machine.

The systems and methods described herein can be embodied and/orimplemented at least in part as a machine configured to receive acomputer-readable medium storing computer-readable instructions. Theinstructions can be executed by computer-executable componentsintegrated with the application, applet, host, server, network, website,communication service, communication interface,hardware/firmware/software elements of a operator computer or mobiledevice, wristband, smartphone, or any suitable combination thereof.Other systems and methods of the embodiment can be embodied and/orimplemented at least in part as a machine configured to receive acomputer-readable medium storing computer-readable instructions. Theinstructions can be executed by computer-executable componentsintegrated by computer-executable components integrated with apparatusesand networks of the type described above. The computer-readable mediumcan be stored on any suitable computer readable media such as RAMs,ROMs, flash memory, EEPROMs, optical devices (CD or DVD), hard drives,floppy drives, or any suitable device. The computer-executable componentcan be a processor, but any suitable dedicated hardware device can(alternatively or additionally) execute the instructions.

As a person skilled in the art will recognize from the previous detaileddescription and from the figures and claims, modifications and changescan be made to the embodiments of the invention without departing fromthe scope of this invention as defined in the following claims.

I claim:
 1. A method for automatically capturing data from non-networkedproduction equipment includes: detecting a first location of a mobiledevice within a facility at a first time during performance of a step ofan augmented digital procedure at a machine in the facility by anoperator; in response to the first location of the mobile device fallingwithin a threshold offset from the machine at the first time, triggeringan optical sensor in the mobile device to capture a first image of themachine; detecting a display on the machine in a first region of thefirst image; detecting a first value, presented on the display atapproximately the first time, in the first region of the first image;and storing the first value in a procedure file for the augmenteddigital procedure performed at the machine.
 2. The method of claim 1:further comprising: detecting a first orientation of the mobile devicewithin the facility at the first time; and estimating a first positionof the mobile device relative to the display on the machine at the firsttime based on the first location and the first orientation; and whereintriggering the optical sensor in the mobile device to capture the firstimage of the machine comprises triggering the optical sensor in themobile device to capture the first image of the machine further inresponse to the first position of the mobile device falling within athreshold distance and a threshold angular offset from the display onthe machine at the first time.
 3. The method of claim 1: whereintriggering the optical sensor in the mobile device to capture the firstimage of the machine comprises triggering the optical sensor in themobile device to capture a sequence of images at approximately the firsttime, the sequence of images comprising the first image and a secondimage; further comprising: detecting a first feature in the first regionof the first image corresponding to a first frame type rendered on thedisplay; detecting the display in a second region of the second image;detecting a second feature in the second region of the second imagecorresponding to a second frame type rendered on the display; and inresponse to detecting the second feature in the second image, discardingthe second image; and wherein storing the first value in the procedurefile for the augmented digital procedure performed at the machinecomprises storing the first value in the procedure file in response tothe augmented digital procedure specifying data capture from the firstframe type rendered on the display.
 4. The method of claim 1: whereindetecting the first location of the mobile device within the facility atthe first time comprises detecting the first location of the mobiledevice worn by the operator at the first time while performing the stepof the augmented digital procedure at the machine; and wherein storingthe first value in the procedure file for the augmented digitalprocedure performed at the machine comprises storing the first value,linked to the step, in the procedure file for the augmented digitalprocedure.
 5. The method of claim 1: further comprising accessing adisplay model associated with the step of the augmented digitalprocedure; wherein detecting the display on the machine in the firstregion of the first image comprises detecting a set of features,analogous to template display features defined in the display model, inthe first region of the first image; and wherein detecting the firstvalue in the first region of the first image comprises transforming theset of features, detected in the first region of the first image, intothe first value according to the display model.
 6. The method of claim1: further comprising retrieving a first data extraction model from aset of data extraction models stored in a database, the first dataextraction model specified in the augmented digital procedure, linked tothe step, and corresponding to a type of the display on the machine; andwherein detecting the first value in the first region of the first imagecomprises: extracting a set of features from the first region of thefirst image; and interpreting the first value from the set of featuresaccording to the first data extraction model.
 7. The method of claim 6,wherein retrieving the first data extraction model from the set of dataextraction models comprises selecting the first data extraction modelfrom the set of data extraction models comprising: a dial-type dataextraction model defining detection of a dial in an image, detection ofa needle within the dial in the image, and interpretation of a valuefrom the image based on a position of the needle relative to the dial; amulti-segment-type data extraction model defining detection of amulti-segment display in an image and interpretation of a value from themulti-segment display; and a digital-type data extraction model definingdetection of a digital display in an image and interpretation of a valuefrom the digital display.
 8. The method of claim 6: wherein retrievingthe first data extraction model comprises retrieving the first dataextraction model defining detection of a dial in an image, detection ofa needle within the dial in the image, and interpretation of a valuefrom the image based on the dial and the needle; further comprisingretrieving a dial mask comprising a set of virtual dial indicatorsassociated with the display on the machine and linked to the step of theaugmented digital procedure; and wherein detecting the display on themachine in the first region of the first image comprises, based on thefirst data extraction model: detecting the display comprising an analogdial in the first region of the first image; and detecting an analogneedle in the first region of the first image; and wherein detecting thefirst value in the first region of the first image comprises: projectingthe dial mask onto the analog dial detected in the first image; andinterpreting the first value from the first image based on a position ofthe analog needle relative to the set of virtual dial indicators in thedial mask.
 9. The method of claim 1: wherein triggering the opticalsensor in the mobile device to capture the first image of the machinecomprises triggering the optical sensor in the mobile device to capturea sequence of images over a duration of time specified in the step ofthe augmented digital procedure, the sequence of images comprising thefirst image; wherein detecting the first value in the first region ofthe first image comprises extracting a sequence of values from regionsin the sequence of images depicting the display; further comprisingaggregating the sequence of values into a timeseries of values based ontimestamps of images in the sequence of images; and wherein storing thefirst value in the procedure file for the augmented digital procedurecomprises storing the timeseries of values in the procedure file. 10.The method of claim 1: wherein triggering the optical sensor in themobile device to capture the first image of the machine comprisestriggering the optical sensor in the mobile device to capture a sequenceof images over a duration of time specified in the step of the augmenteddigital procedure, the sequence of images comprising the first image;wherein detecting the first value in the first region of the first imagecomprises extracting a sequence of values from regions in the sequenceof images depicting the display; further comprising calculating rates ofchange across values in the sequence of values; and wherein storing thefirst value in the procedure file for the augmented digital procedurecomprises storing the first value following a segment of the sequence ofvalues exhibiting less than a threshold rate of change.
 11. The methodof claim 1, further comprising, in response to detecting the first valuein the first region of the first image: generating a notificationconfirming data capture from the machine; at the mobile device,presenting the notification to the operator; and closing the step of theaugmented digital procedure.
 12. The method of claim 1: furthercomprising, at the mobile device, capturing a video feed via the opticalsensor during the augmented digital procedure; wherein detecting thefirst location of the mobile device within the facility at the firsttime comprises: extracting a constellation of features from the videofeed; and calculating the first location and a first orientation of themobile device within a localization map of the facility that aligns theconstellation of features to corresponding features represented in thelocalization map; and further comprising: projecting a field of view ofthe optical sensor into the localization map based on the first locationof the mobile device, the first orientation of the mobile device, andoptical characteristics of the optical sensor; and predicting presenceof the display in the field of view of the optical sensor in response tothe field of view of the optical sensor, projected onto the localizationmap, containing a location of the display represented in thelocalization map; and wherein triggering the optical sensor in themobile device to capture the first image of the machine comprisestriggering the optical sensor in the mobile device to capture the firstimage of the machine in response to predicting presence of the displayin the field of view of the optical sensor at the first time.
 13. Themethod of claim 1, further comprising, during a second step of theaugmented digital procedure: rendering a video feed captured by theoptical sensor on a viewfinder of the mobile device; accessing a virtualguide associated with the second step of the augmented digitalprocedure; rendering the virtual guide over the video feed on theviewfinder; prompting the operator to align the virtual guide, renderedon the viewfinder, with a second display on the machine; recording asecond image in response to detecting alignment between the virtualguide rendered on the viewfinder and a second display on the machine;detecting a second value, presented on the second display, in the secondimage; and storing the second value, associated with the second step, inthe procedure file for the augmented digital procedure performed at themachine.
 14. The method of claim 1: wherein storing the first value inthe procedure file for the augmented digital procedure comprises storingthe first value as a verification value, linked to the step in theaugmented digital procedure, in the procedure file; and furthercomprising: at the mobile device, prompting the operator to manuallyrecord a reference value for the step according the augmented digitalprocedure; storing the reference value, linked to the step in theaugmented digital procedure, in the procedure file; and in response to adifference between the verification value and the reference value,flagging the procedure file for review.
 15. The method of claim 1:further comprising: accessing a document specifying a set of steps of aprocedure for an equipment type of the machine, the set of stepscomprising the step; linking a first description of the step in theprocedure, extracted from the document, to a first input field;assigning a first capture type to the first input field; generating adigital draft procedure comprising a set of descriptions of the set ofsteps extracted from the procedure and the first input field specifyingcapture of data of the first capture type from the equipment type; at aninitial time preceding the first time, serving the digital draftprocedure to a second mobile device, assigned to a second operator, forcompletion at the machine; defining a first spatial trigger forcapturing data from the display on the machine during the step based ona position of the second mobile device during completion of the step inthe digital draft procedure by the second operator; generating theaugmented digital procedure comprising the set of descriptions of theset of steps, comprising the first spatial trigger for capturing datafrom the display during the step, defining the first capture type forthe step, and linking the first spatial trigger to the first capturetype and the first input field for the step; and following the initialtime, serving the augmented digital procedure to the mobile device;wherein selecting the first image comprises selecting the first image inresponse to proximity of the first location of the mobile device to thefirst spatial trigger during completion of the step in the augmenteddigital procedure; wherein detecting the first value in the first regionof the first image comprises extracting the first value from the firstregion of the first image according to the first capture type; andwherein storing the first value in the procedure file for the augmenteddigital procedure comprises storing the first value, linked to the firstinput field, in the procedure file.
 16. The method of claim 1: whereinstoring the first value in the procedure file comprises storing thefirst value, linked to the step of the augmented digital procedure, inthe procedure file; and further comprising: detecting a second locationof the mobile device within the facility at a second time during asecond step of the augmented digital procedure; in response to thesecond location of the mobile device falling outside of the thresholdoffset from the machine at the second time, accessing a second imagecaptured by a fixed camera at approximately the second time, the fixedcamera arranged in the facility and defining a second field of viewintersecting the display; detecting the display on the machine in asecond region of the second image; detecting a second value, presentedon the display at approximately the second time, in the second region ofthe second image; and storing the second value, linked to the secondstep of the augmented digital procedure and associated with the secondtime, in the procedure file.
 17. A method for automatically capturingdata from non-networked production equipment includes: accessing a firstimage captured by an optical sensor within a facility at a first timeduring performance of a step of an augmented digital procedure at amachine in the facility by an operator; in response to the step in theaugmented digital procedure specifying data capture from a display ofthe machine, accessing a visual display feature representing thedisplay; detecting the display in a first region of the first imagebased on the visual display feature; extracting a first value, presentedon the display at the first time, from the first region of the firstimage; and storing the first value in a procedure file for the augmenteddigital procedure performed at the machine.
 18. The method of claim 17,wherein accessing the first image comprises accessing the first imagecaptured by the optical sensor integrated into a mobile device worn bythe operator during the augmented digital procedure at the machine. 19.The method of claim 17: further comprising retrieving a first dataextraction model from a set of data extraction models stored in adatabase, the first data extraction model specified in the augmenteddigital procedure, linked to the step, and corresponding to a type ofthe display on the machine; and wherein detecting the first value in thefirst region of the first image comprises: extracting a set of featuresfrom the first region of the first image; and interpreting the firstvalue from the set of features according to the first data extractionmodel.
 20. A method for automatically capturing data from non-networkedproduction equipment includes: accessing a first image captured by anoptical sensor within a facility at a first time during performance of astep of an augmented digital procedure at a machine in the facility byan operator; in response to the step in the augmented digital procedurespecifying data capture from a display of the machine, accessing a dataextraction model, from a set of data extraction models, definingextracting of data linked to the step of the augmented digital procedurefrom images of the display; detecting the display in a first region ofthe first image based on the visual display feature; extracting a firstvalue, presented on the display at the first time, from the first regionof the first image based on the data extraction model; and storing thefirst value in a procedure file for the augmented digital procedureperformed at the machine.